How Do ASH Guidelines Panels Make Decisions? Association between Decision Making Factors and the Strength of Recommendations

Background: Clinical practice guidelines (CPGs) represent a key mechanism for optimizing health care decision making. CPGs are a product of discussion by a group of, typically, 10-20 individuals with varying expertise. Little is known how CPG panels actually make their recommendations.

Objective: In a study of real-life decision making, we investigated the factors considered by members of panels convened by the American Society of Hematology (ASH) to develop guidelines, when using the widely accepted formal GRADE (Grading of Recommendations Assessment, Development, and Evaluation) system.

Methods: To account for panel level factors and individual level factors, we employed two level hierarchical, random-effect, multivariate logistic and ordered logistic regression analysis.

Results: 101 participants taking part in 8 CPGs panels issued 1,289 recommendations. Association of GRADE (normative) factors with the strength of recommendations (SOR) dominated the findings over the non-GRADE (descriptive) factors. In the main analysis certainty in evidence (regardless of direction for or against intervention) [OR=1.83 (95CI% 1.45 to 2.31;p<0.0001)], balance of benefits and harms [OR=1.49 (95CI% 1.30 to 1.69;p<0.0001)] and variability or uncertainty in the patients' values and preferences [OR=1.47 (95CI% 1.15 to 1.88;p<0.002)] were the strongest predictors of SOR coded as "neither for nor against" , "weak for or against" or "strong for or against" health intervention. Greater judgment of certainty of evidence proved highly associated with a strong recommendation [OR=3.60 (95% CI 2.16 to 6.00;p<0.00001] when panel members were issuing recommendation "for" interventions. When, however, panels made recommendations "against" intervention, certainty in evidence was not associated with probability of issuing strong recommendation [OR=0.98 (95%CI: 0.57 to 1.8; p=0.94)]. Two panelist characteristics were associated with strong recommendations: age (per decade) [OR=1.79 (95 CI% 1.2 to 2.84; p<0.005)], and greater intolerance of uncertainties [OR=0.57 (95 CI% 0.37 to 0.86; p<0.008)]. Agreement between individual panel members and the group ranged from very poor (average kappa of -0.01 in one panel) to moderate (kappa=0.64 in another panel), with most panels in an intermediate range. We also found that the panel members who were asked by the ASH to recuse themselves from voting due to high risk of conflict of interest (COI) would have voted differently if they were allowed to do so.

Conclusion: Factors associated with GRADE's conceptual framework proved, in general, highly associated with strong versus weak recommendations and with the direction of recommendation. However, some non-GRADE factors of importance for decision-making were also identified. Findings that panel members with high risk of COI made different judgments than those without COI provide empirical support for the importance of managing conflict of interest. The low agreement between individual panel members and group consensus, and failure of certainty of evidence to be associated with strength of recommendations against an intervention, suggest the need for improvements in the process.